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Abstract 



In this paper we consider the class of anti-uniform Huffman codes and 
derive tight lower and upper bounds on the average length, entropy, and 
redundancy of such codes in terms of the alphabet size of the source. The 
Fibonacci distributions are introduced which play a fundamental role in 
AUH codes. It is shown that such distributions maximize the average 
length and the entropy of the code for a given alphabet size. Another 
previously known bound on the entropy for given average length follows 
immediately from our results. 

1 Introduction 

Consider a discrete source with finite size alphabet S = {si, S2, ■ ■ ■ , s n } and 
associated ordered probability distribution V = (pi,p n > ■ ■ ■ ,Pn) where pi > 
p 2 > . . . > p n . It is well-known that the Huffman encoding algorithm [1] 
provides an optimal prefix-free code for this source. A binary Huffman code 
is usually represented using a binary tree T, whose leaves correspond to the 
source symbols; The two edges emanating from each intermediate node of T arc 
labeled with cither and 1, and the codeword corresponding to a symbol is the 
string of labels on the path from the root to the corresponding leaf. Huffman's 
algorithm is a recursive bottom- up construction of T, where at each time the two 
smallest probabilities are merged into a new unit, and henceforth represented 
by an intermediate node in the tree. 

We denote by li the length of the codeword associated to symbol Si which is 
the number of edges from the root to the node Si on the Huffman tree. Then, 
the expected length of the Huffman code is defined as 



n 




(1) 



Similarly, the entropy of the source is defined as 
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(2) 



i=i 
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Figure 1: An anti-uniform Huffman tree for a source with n symbols. 



where all the logarithms in this paper are in base 2. The Huffman encoding is 
optimal in the sense that no other code for distribution V can have a smaller 
expected length than L(V). The redundancy R(V) of the code is defined as the 
difference between the average codeword length L(V), and the entropy H(V) 
of the source. It is easy to show that the redundancy of the Huffman code is 
always non-negative and never exceed 1. 

In contrast with uniform Huffman code wherein |Zj — lj\ < 1, a code (source) 
is called anti-uniform Huffman (AUH) jFj (Fig[l} if 1^ = i for i = 1, . . . , n— 1 and 
l n = n — 1. Such sources can be generated by several probability distributions. 
It has been shown in [2] that the normalized tail of the Poisson distribution 
satisfies AUH structure. These kinds of distributions arc also considered by 
Kato et. al. [3] and in particular it is shown that the geometric distribution 
with success probability greater than some critical value satisfies AUH condition. 

The class of AUH sources are also known for their property of achieving 
minimum redundancy in different situations. It has been shown in [5] that 
AUH codes potentially achieve the minimum redundancy of Huffman code of 
a source for which the probability of one of the symbols is known. A similar 
result by Capocelli et. al. [3] shows that AUH structure achieves the minimum 
redundancy of a Huffman codes when p n , the probability of the least likely 
symbol is known. 

In this paper we consider the AUH structure and obtain tight bounds on 
the average codeword length, entropy and redundancy of such codes in terms 
of n, the alphabet size of the sources. The rest of the paper is organized as 
follows. We will start with a useful lemma in the next section. Then we state 
and prove our bound on the average length, entropy and redundancy of AUH 
codes in Sections [31 and respectively. Finally we conclude in Section [U 
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Figure 2: Decomposition of a Huffman tree around an intermediate node u. 



2 Preliminaries 

One can simply define the probability of an intermediate nodes on the Huffman 
tree as the sum of the probabilities of the leaves lying under it. In an AUH tree 
of a source with n symbols in Fig[TJ there arc n — 2 intermediate nodes which 
are labeled by qi, . . . , q n -2- We denote the part of a Huffman tree lying under 
any intermediate node, u, by A u (see Figf2]). It is clear that A u is a subtree 
which satisfies the Huffman structure, unless the probability of the root is not 
one. So by normalizing the probabilities of all the leaves by u, the probability 
of the intermediate node, we obtain a new Huffman tree which is denoted by 
it" 1 * A u . On the other hand, we can merge all the leaves lying in a subtree 
A u in T and obtain a new Huffman tree which is denoted by A u . The following 
lemma [5] relates the parameters of a source and the its corresponding tree to 
the parameters of its subtrees. 

Lemma 1. For any intermediate node with probability u, 

H(T) = H(A U ) + uHiu- 1 * A„). (3) 
The same equation holds holds for average length and redundancy. 



3 Average Length 

The average length of any non-trivial code is lower bounded by 1. Using Lemma, 
[l] it can be shown that for any arbitrary n the average length of the AUH source 
with distribution 



' n < £ \ ' 2 ' 4' ' ' ' ' 2™~ 3 ' 2™ -2 ' 2 n ~ 2 

tends to 1 as e — > 0. Therefore, average length of an AUH code is tightly 
lowerbounded by 1. 

In the following we will state a tight upperbound on the average length of 
AUH codes in terms of alphabet size of the source. A similar result is also shown 
independently in the upcoming paper [7]. 



Theorem 1. Let V be a distribution over a discrete source of alphabet size n. 
Then L(V) is upperbounded by 

LT X = (4) 

Jn+l 

where f n is the n-th Fibonacci number defined as f\ = f% = 1 and 

fn = fn-l + fn-2 U > 3. (5) 

Furthermore, this bound is tight and can be achieved by the Fibonacci distribution 

T>(F) _ f fn-l fn-2 $3 fl fl ^2 

\Jn+l Jn+l Jn+l Jn+l Jn+l Jn+l 

Before stating the proof, we show two simple lemmas which simplify the 
proof. 

Lemma 2. In any probability distribution V = (pi,p2, ■ ■ ■ ,p n ) which maximizes 
the average length, the probability of any arbitrary leaf is not grater that the 
probability of the intermediate node in the same level, i.e., 

Pi <q, i=l,...,n- 2. 

where q. t = V _ , /, ,. 

Proof. Let pt > qi for some i. This implies e = (pi — qi)/2 is positive. Defining 
Si = —e and eu = Ph^/li for k > i, we can show the distribution 

V = (pi, . . . ,Pi-l,Pi + EhPi+1 + £»+l, ■ ■ ■ ,Pn +£n) 

satisfies the AUH constraints and 

n-l 



L(V')-L(V) = J2-kk + e n {n-l) 



k—i 
n-l 



= s k (k - i) + e n (n - 1 - i) >0 

k=i+l 

which is in contradiction with the maximality of "P. □ 

Lemma 3. Any probability distribution V = {pi,P2i ■ ■ ■ iPn) with maximum 
average length, satisfies 

Pi = Q2 = ^2p3- (6) 

i>2 

Proof. The structure of the Huffman tree and LemmaJ5] imply p\ > (72 > P2- 
Assume the LHS inequality is strict and so e = (pi — 92)/2 is positive. Defining 
£1 = —£, £k = Pk£/Qi for fc > 1, and 

V = (pi +£l,P2 + £2, ■ ■ ■ ,Pn + £n) , 
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we have 

n-l 

L(V')-L(V) = ^e fc fc + e„(n-l) 

k=l 
n-l 

= ^£ fe (fc-l) + e„(n-2) >0 

which refuses the maximality of V . □ 

Proof of Theorem\]\ We proof the theorem using induction over the alphabet 
size, n. It is clear that L™ ax = 1 = (f$ — 3)//3. For n = 3, one can argue that 
= (1/3, 1/3, 1/3) has the maximum average length L™ ax = (/ 6 — 3)//4 = 5/3. 
Let the theorem is true for any k < n, and 7-* = (pi,p2, ■ ■ ■ ,Pn) achieves the 
maximum average length of an AUH for n symbols. We consider two case as 
follows. 

(i) Pi > f+i • We denote the subtree lying under qi = i — pi by Ai_ Pl as 

before. Using LemmaflJ we have 

L(P) = l + (l-p 1 )L((l- Pl )- 1 *A 1 _ Pl ) 
< x + (1 — ) f n + 2 ~ ^ 

/n+1 fn 

/n+3 ~ 3 
/n+1 

where the inequality follows from the assumption of the induction for 
n = k — 1. 

(ii) Pi < /"+i • Using Lemma. Owe have q 2 = p\. By expanding L(V) with 

respect to qi and using Lemma, [I] we can write 

L(T) = L(A q2 ) + q 2 L(q^ * A q2 ) 
= 2 + Pl (L( P ^ * A 92 ) - 1) 

< 2 +^( / "+ 1 ~ 3 -l) 

J n+1 Jn—1 
/n+3 ~ 3 
/n+1 

where the inequality is the assumption of the induction for n = k — 2. 

□ 

Remark. 1 One can simply show that meets the upper bound for any alpha- 
bet size. Although some other distributions such as P4 = (0.35,0.30,0.20,0.15) 
can meet the bound, it can be shown that the maximal distribution is unique 
for n > 4. 

Remark. 2 Note that the Fibonacci probability distribution, tends to 

asm 00, where t = g -1 is the positive root of x 2 + x — 1 = 0. Furthermore, 
It is easy to see that {L™ ax }^ =1 is an increasing sequence and tends to t~ 2 = 
3+ 9 — 2.618 in the asymptotic case. 
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4 Entropy 



Since only very particular sources satisfy the AUH structure, the range of the 
entropy of such sources is not so wide. It is easy to check that the minimum 
entropy of such sources can be arbitrary close to zero for any alphabet size n. 
In order to see this, one may compute the entropy of 

EE SEE 

P =fi_ f : : 

n ' E v ' 2' 4' " ' ' 2™~ 3 ' 2™~ 2 ' 2™~ 2 

for e < 2/3 and show H(P n£ ) — > as e — > 0. 

In spite of that, uppcrbounding the entropy of AUH codes is not trivial. It 
has been shown in [6] that the entropy of an infinite length AUH codes with 
given average length L is upper bounded by 

H™*{L)=L\ogL-(L-l)\og{L-l). (7) 

This bound is only valid for infinite source. It can be shown that any dyadic 
source (p = 2~ l for some integer I) has entropy larger than H™ ax (L). The 
following theorem states a tight upperbound on the entropy of AUH sources 
with n symbols. 

Theorem 2. The entropy of finite source with n symbols is upperbounded by 

n— 1 

Hr x = H{P<p) = log/„ +1 - — - J2 filogfi- (8) 

The proof of this theorem is fairly similar to that of Theorem|TJ The fol- 
lowing Lemmas show some basic properties on distributions which achieve the 
maximum entropy. 

Lemma 2'. Let V = (jpi,p%, ■ ■ ■ ,p n ) be a distribution over n symbols with 
maximum Entropy. Then pi < qifor any i = 1, . . . , n — 2. 

Proof. Similar for proof of Lcmmal2 we assume that the condition does not 
hold for some i and make a contradiction. Assume e = (p, — qi)/2 > 0, and 
define the modified distribution 

P' = (Pi, ■ ■ ■ ,Pn) = (Pl,P2, ■ ■ ■ ,Pi-l,Pi + £i,Pi+l + Si+l, ■ ■ -,Pn+£n) , 

where £; = — £ and = pk£/qi for k > i. We can write 
H{P')-H{T) = Y.P^ogPk-Y.P>Sp' k 

k k 

= Yl Pk l0g Pk ~ X! Pk l0g P 'k ~^2(Pk + £k) log p' k 

k k<i k>i 

= £(p||^)+]r £fc iog4>o 

k>l Pk 

where -D(-||-) is the Kullback-Lcibler divergence and the last inequality follows 
from the facts that ^2 k<i £k = — £j and V' is an decreasing sequence. This 
inequality is in contradiction with assumption, which implies the desired result. 

□ 
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Lemma 3'. For any distribution V = (pi,p2, ■ ■ ■ >Pn) which achieves the maxi- 
mum entropy, p\ = q 2 . 

Proof. The structure of AUH tree implies pi > q 2 . If the inequality is strict, by 
Lemma. 2' we have, p\ > q 2 > p 2 . Define s = (pi — q 2 )/'2 > 0, and consider the 
distribution 

V = (p[, . . . ,p' n ) = (pi +£i,P2 +e 2 , ■ ■ ■ ,p n +e n ) , 
where E\ = —s and Ek = Pk£/qi for k > 1. We have 

H(T')-H(T) = ^Tpfclogpfc-^^log^ 

k k 

= ^Pk log Pk-^2 Pk l0g(pfe + £fc 
k k 

= D(V\\V') + y / e k \og?Jf>0 

kTi p * 

which refuses the assumption we made in the beginning. □ 

Proof of Theorem^ Similar to the proof of Thcorcmfll we prove this theorem 
by induction over the alphabet size of the source. Since the uniform distribution 
satisfies the AUH constraints for n — 3 and n — 2, we have L™ ax = 1 and 
L™ = log 3 which coincide with JHJ). Now, let n > 4 and the bound is valid for 
k < n. We consider two cases. 

(i) p > : Using Lemma. [T]and by expansion the entropy with respect to 
<7i, we have 

H(V) = hipt) + qi H(q^ * A qi ) 
< MPi) + (l-Pi)^ a i 

Jn+1 jn+1 
1 n— 1 

= Iog/ n+1 - — J>log/j (9) 
Jn+1 l=l 

where the first inequality comes form the assumption of induction for 
k = n — 1 and the second inequality follows from the fact that the function 
a(x) = h(x) + (1 — x)H™f is non-increasing for x > f n -i/fn+i- It can 
shown by the taking the derivative of a(x) as 

da - Inn- 1 ~ p w max <r 1nn . 1 ~ fn-i/ fn+i „ max 
— - log Vi ^ l0 § — 7 77 H n-1 

dx p fn-l/jn+l 



j, ^ n — 2 

log log /„ + — fi 1°S fi 

Jn — 1 Jn 



l 

fn 



n—2 j, 

y^/ilog-r 5 l°g/n- 

T— f /n-1 



< 0. 
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(ii) p < y + l '■ Using Lemma. 3', we can only focus on the distributions for 
which q2 = Pi- Now we use Lemma. [T]to expand the entropy with respect 
to q 2 . 

H(V) = h(A q2 )+q 2 H(q^*A q2 ) 

= -2p\ogp - (1 - 2p)log(l - 2p)+pH{q^ * A ?2 ) 
< -2p logp - (1 - 2p) log(l - 2p) + pH™l 



.fn—1 fn—1 ri fn — l\- i ( i n fn—l\ , fn— 



< _ 2 ^z±log^±-(l-2^)logll-2^j+^^_ 2 

jn+1 Jn+1 Jn+1 \ Jn+1 / Jn+1 



rmax 



^ 71—1 

= log/„ +1 -- V/ 4 log/, (10) 

Jn+1 l=1 

where again the assumption of induction for k = n — 2 implies the first 
inequality and the second one follows from the fact that f3(x) = —2x logx— 
(1 — 2x) log(l — 2x) + xH^2 is an increasing function for x > f n ~i/ fn+i- 

□ 

Corollary. Note that the maximum achievable entropy for an infinite size al- 
phabet is 

Hm H^ x = ~ £ ^g** = (i + ^) lo S 7 ( n ) 
which is obtained for "Poo and coincides with 

LsriogLsr - (^ ax - 1) io g (^r x - 1) = ^io g l - (1 - 1) i og (l - 1) 

iV 1 

1 + - log-. 



t 2 / "° i 



This simply proves ([7|). 



5 Redundancy 

It is known that the Huffman code associated to any dyadic source has zero 
redundancy and since such distributions exist for any arbitrary alphabet size 
and satisfy the AUH constraints, the redundancy of AUH code is tightly lower- 
bounded by zero. On the other extreme, it can be shown that the redundancy 
of AUH codes can be arbitrary close to 1. 

Theorem 3. For and alphabet size n and 5 > 0, there exist probability distri- 
butions on n symbols for which 

R{V) > 1 - 6. 

Proof. Take an arbitrary AUH distribution Q = (qi, . . . , q n ~i) over n — 1 sym- 
bols and £ > small enough such that I — e > maxjgie, (1 — qi)s}. Therefore, 
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V = (1 — e, <?ie, . . . , g„_i£) is a distribution with AUH code. Using LemmaO]we 
have 

R(T) = l-h(e) + eR(s~ 1 * A e ) 
= l-h{e)+eR(Q) 

where h(x) — —a; log a; — (1 — x) log(l — x) is the binary entropy function. Note 
that R(Q) is bounded by L(Q) < L™\ < t' 2 , and R{T) tends to 1 as e -> 0. □ 

6 Conclusion 

In this paper we have obtained tight upper and lower bounds on the average 
length, entropy and redundancy of the Huffman code for an anti-uniform Huff- 
man source. We showed that for a given alphabet size, Fibonacci distributions 
maximize the average length and entropy. 
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